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ABSTRACT 

The Napa Project is a demonstration research study 
developed to evaluate the effectiveness of seven school-based 
substance abuse prevention strategies. Various programs \iere offered 
to over 7,000 students in grades three through nine. Four programs 
were delivered to students by regular classroom teachers and two 
programs were offered as elective courses. These six programs did not 
address the topiv" of substance abuse, focusing instead on factors 
believed to underlie substance abusa (self-esteem and school 
attitudes). The seventh program was a drug education course which 
taught students relevant competencies and provided information about 
the conseauences of drug use. Thirteen studies were conducted to 
evaluate the effects of the individual programs as well as the 
cumulative effects of several programs delivered to students over a 
2- to 3-year period. The results revealed that the drug education 
course had some positive effects on students* drug knowledge and drug 
involvement, and their perceptions of peers* attitudes toward drugs 
and drug use. These effects occurred primarily for girls, and they 
did not replicate across studies. None of the other programs was 
found to be effective. The results of these studies question the 
value of "generic" approaches to substance abuse prevention. This 
report describes the strategies and discusses the designs, results, 
and implications of the various studies. Reports on the individual 
studies are listed in the appendix. (NB) 
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The Napa Project: Executive Summary 
Pacific Institute for Research and Evaluation 



The Napa Project was funded by thtj National Institute on Drug Abuse to 
evaluate seven school-based substance abuse prevention programs. The programs 
were consistent with a prevention approach that focused on improving school 
and peer group influences along with student attitudes and competencies. 
Four programs were delivered to students by regular classroom teachers who 
had received special in-service education, and two programs were offered as 
elective courses. These six "generic" programs did not address the topic of 
substance abuse; rather, they focused upon factors believed to underlie substance 
abuse (e.g., self-esteem, attitudes toward school). The final program, a drug 
education course, taught students relevant competencies and provided information 
about the consequences of drug use. The specific programs were: 

• Magic Circle--teachers were prepared to lead structured small- 
group discussions on particular topics (grades 3-4); 

• Effective Classroom Management-Elementary— teachers were trained 
in communication skills, discipline techniques, and self-concept 
enhancement techniques (grades 4-6); 

• Effective Classroom Management-Junior High— above program adapted 
for junior high environment (arades 7-9); 

• Jigsaw--teachers were trained to organize students into small 
learning groups in which each student taught part of the regular 
curriculum to other group members (grades 4-6); 

• Cross-Age Tutoring— elective course in which students learned 
teaching skills and tutored younger children in academic subjects 
(grades 8-9); 

• Operating a School Store— elective course in which students operated 
a store on their campus and learned relevant business skills (grades 
8-9); 

• Drug Education— students learned a systematic decisi on ^making process, 
a framework for understanding their needs and motives, commercial 
advertising techniques, assertiveness skills for dealing with peer 
pressure, and were provided information about drugs vgi^ad^s /-oj. 
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The research was conducted in the Napa Valley (CA) Unified School District. 
Over 7000 students participated in 13 studies that evaluated the effects of the 
individual programs as well as the cumulative effects of several programs 
delivered to groups of students over a two- to th»"ee-year period. In each 
study data were collected at pretfst, posttest and foUcw-up from a control 
group as well as from the group that received the program. School -related 
outcomes were assessed, such as self-esteem and academic achievement, in 
addition t 'rug-specific outcomes, sucn as drug attitudes and drug use. The 
imple.Ttrit ♦ lo. of the programs was assessed throuc.i student or teacher ic'orts 
and classroom observations. 

The drug education course had some positive effects on participants' drug 
knowledge and drug involvement, and their perceptions of peers' attitudes 
toward drugs and drug use. However, these effects occurred primarily for girls, 
and they did not replicate across studies. None of the othc^ programs was 
found to be effective. The results call into question the value of "generic" 
approaches to substance abuse prevention. 

Th2 policy implications of this project are important. The project shows 
that rigorous evaluation of prevention programs is indeed feasible and worthwhile, 
and it suggests many ways of improving evaluation methooo. It also shows that 
generic approaches may have little usefulness as prevention techniques, at 
least as they are being implemented by many programs today. 
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INTRODUCTION 



This report describes the Napa Project, a demonstration research study 
funded by the National Institute on Drug Abuse. The primary purpose of the 
project was to evaluate the effectiveness of seven school-based substance 
abuse prrvention strategies. The project sought to answer the question, 
"What are the effects of these promising scho ol-based prevention strategies 
when they are intensively and carefully implemented under favorable circum- 
stances?" 

One or more separate evaluations of each strategy were conducted during 
the course of the project. In other studies, called "Cohort" studies, two or 
three strategies were provided to the sarr.e group of students over two- or 
three-year periods. In the Cohort studies, the cumulative effects of the 
strategies were measured each year. All studies employed experimental or 
quasi-experimental designs in which students who received the strategies v/ere 
ccnpared with students wno did not. 

The following sections describe the strategies and discuss the designs, 
results, and implications of the various studies. Reports on the individual 
studies are listed in the appendix. 



3. 



INTERVENTIONS 

Background 

In the 1970's, affective education and "alterndti ves"^ programs became 
increasingly popular as substaic^ abuse prevention strategies. The rise of 
these "generic"- strategies accompanied the decline in popularity of drug- 
specific informational approaches which were heavily laden with "scare 
tactics." Affective education was usually justified on the basis of the 
numerous correlational studies that found an association between social 
co mpetencie s , self-attitudes and values, and drug abuse (e.g., Ahlgren and 
Norem-Hebeisen, 1979; Smith and Fogg, 1978). It was generally believed that 
these attitudes and behaviors precede and thereby mediate drug abuse. There- 
fore, many affective education and alternatives strategies have focused on 
teaching intrapersonal and social conpetencies to children, and on creating 
environments that are responsive to children's emotional and social as well 

as their cognitive needs. 

The curricula of affective education programs include self-esteem building, 

interpersonal skill development, and decision-making/problem-solving techniques. 

They also include methods for restructuring academic classroom activities in 

ways thought to promote learning and positive relationships. Common to the 



^Alternatives are defined as "constructive involvements that act as mean- 
ingful options to drug and alcohol use (Schaps and Slimmon, 1975)," and are 
based on the hypothesis that drug abuse can be prevented by providing more 
fulfilling experiences and activities. 

^The terms "generic," "indirect," and "non-specific" all refer to pre- 
vention strategies which do not directly address the topic of substance abuse. 
Instead, they focus upon factors believed to underlie drug abuse and other 
problem behaviors. 
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various approaches is the assumption that teachers should be responsive to 

the affective as well as cognitive needs of students. Thus, teacher in-service 

Is a staple of affective development programs for students. 

The teaching of affective skills has been recently introduced in school- 
based drug education programs. These programs provide information about 
drugs and teach skills for utilizing and acting on the information (e.g., 
decision-making and assertion skills). Recent research on the prevention of 
cigarette smoking suggests that such courses may have lasting efrects (Arkin et al., 
1981; Botvin and Eng, 1982; Botvin et al., 1980; Evans et al., 1981; Flay et al., 
1983; Hurd et al., 1980; McAlister et al., 1980; Perry et al., 19S0). 

Strategies 

Each strategy selected for study met several criteria. First, each was 
consistent with a general approach that emphasizes school and peer group 
influences, and individual competencies and attitudes. Each was representative 
of current school-based prevention programs in affective education, alternatives 
programs, or drug education. Each could be implemented at moderate cost and 
without major changes in the priorities and constraints under which most public 
schools operate. Finally, each strategy could not involve serious physical 
or psychological risks to participants. 

Where possible, established, "packaged'* strategies were selected. Other- 
wise, project staff developed the curriculum for a strategy using elements 
from existing curr. '.a and program materials. 

Four of the strategies were in-service teacher training courses t.iat 
focused upon classroom and individual factors thought to influence attitudes 



ERIC 



9 



5. 



toward school, self-esteem, and the development of social competencies.. None of 
the courses addressed the topic of drug use. The in-service teaching training 
strategies were: 

• Magic Circle— teachers were prepared to lead structured small- 
group discussions on particular topics in their classrooms 
(grades 3-4); 

• Effective Classroom Management-Elementary (ECM-Elementary) -- 
teachers were taught various communication skills, discipline 
techniques, and self-concept enhancement techniques (grades 4-5); 

• Effective Classroom Management-Junior High (ECM-JH)--communication , 
discipline, and self-concept enhancement skills were adapted for 
teaching in the junior high environment (giJdes 7-9); 

• Jigsaw--teachers were taught to organize classrooms into learning 
groups of five or six students in which each student teaches an 
essential piece of the regular curriculum to the other group 
members (grades 4-6). 

Two alternatives were offered as elective academic courses to junior 
high school students. In the courses, students were taught skills and provided 
opportunities for helping peers or younger children. The courses did not 
address the topic of drug use; instead, they sought to strengthen construc- 
tive self-concepts and to teach social competencies. The alternatives 
strategies were: 

• Cross-Age Tutori ng--students tutored younger children on a 
regular basis in reading or other academic subjects (grades 8-9); 

f Operating a School Store--students ran a school store on their 
camous, selling school supplies and snacks, while learning 
relevant business skills in a related academic course (grades 8-9). 

The final strategy was a drug education course that taught drug information 

and social competencies to seventh graders. In the final version of the course, 

students were taught Maslow's (1980) framework for understanding motivation; 

analyzed techniques used in commercial advertising; learned a systematic 
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decision-making process; and learned asserti veness skills for dealing with 
peer pressure. Towards the end of the course, students were provided informa- 
tion about psychoactive substances with emphasis on tobacco, alcohol, and 
marijuana. Students also applied the social competencies in considering drug 
use issues . 

Implementation 

Process data gathered from students and teachers during initial imple- 
mentations of the strategies were used to revise curricula and procedures for 
subsequent implementation. The ECM and Drug Education strategies were 
substantially revised, ano the other strategies were modified in minor ways. 
Unless otherwise indicated, the following descriptions refer to the final 
versions of the strategies. 

Teachers were trained in each in-service strategy through 9 to 12 weekly 
two-hour workshops. Several times during and after training, the trainer observed 
each teacher's use of the in-service skills in the classroom, providing additional 
encouragement and guidance. All of the in-service courses combined lecture, 
discussion, readings, simulations and practice exercises. At each training 
session, previously taught skills were reviewed, implementation problems were 
discussed, and new skills were introduced and practiced. All teachers who com- 
pleted the training received a stipend, and graduate credit was offered. 

The first version of the Cross-Age Tutoring course was offered each serester 
to eighth and ninth grade students, and was taught by a junior high school 
teacher assisted by project staff. The second course was offered to eighth 
grade students only, and was taught by project staff. The class met daily 
during the entire semester. Tutors traveled to nearby elementary schools to 
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work one-on-one or in small groups v.'ith younger students. They also met as 
a group to refine skills, discuss problems, and plan schedules. Project 
staff closely monitored the tutors' activities at the elementary schools. 
Tutors received grades and academic credit for their participation in the 
course. 

The School Store class met daily and was taught by a junior high teacher 
with assistance from project staff. Teaching methods included lecture, 
demonstration, self-guided learning modules, experiential activities, simula- 
tions, and role-playing. Students volunteered some of their own time to work 
in the store. Each student participated in most aspects of store operations, 
including sales, marketing and accounting. 

The 12 sessions of the Drug tvlucation course were taught by a project 
staff member onci per week in social studies classes. Instruction included 
.^cture, demonstration, experiential activities, role-playing and audio-visual 
presentations. 
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EVALUATION METHODS 

The strategies were evaluated individually and in combinations 

in 12 stuQies. All studies assessed the implementation of the strategies 
as wen as their effects upon students. The methods used in conducting 
the process and outcome evaluations are described in this section. 

Process Evaluation 

In each study, process data were gathered to monitor implementation of 
the st**ategy and to assess participants' reactions. 

In monitoring the in-service training sessions, teachers' attendance at 
each session was recorded; anonymous teacher ratings of each session were 
collected; teacher participation in the sessions was observed; and the agenda, 
content, and procedures of each session were documented. At the end of the 
training and at the end of the school year, the teachers were surveyed 
regarding their overall assessments of the course, the trainers, and the 
follow-up classroom visits by the trainers. Frequency and quality of class- 
room implementation of the strategies were monitored by: surveys of the 
teachers at mid-year and year-end; the trainer's classroom observations; 
classroom observations conducted by project research personnel; and in the 
case of Magic Circle and Jigsaw, weekly implementation logs provided by 
teachers 

Participation in the alternatives strategies was monitored through 
trainer's observations of class sessions, and through observations of students 
during tutoring sessions and when working in the school store. Also, students 
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were surveyed at the end of each semester regarding their evaluations of 

the courses and their assessments of the tutoring or schcol store experiences. 

The procedures ard content of the drug education course were documented, 
and three surveys of students' reactions to the course were conducted after 
the fourth, seventh, and final sessions. 

Outcome Evaluation 

Table 1 summarizes the research designs for each of the 13 evaluation 
studies. The letters following each study identify the rele" ant reports 
as listed in the appendix. Table 1 shows: 

# Average number of students per group used to assess main effects 
of interventions ; 

# Major testing and intervention activities in each project year; 

# Students' grade level in each year of the study. 

Also listed in Table 1 is an annual drug survey, administered to large samples 
of junior and senior high school :tudents each spring. 

Outcome evaluations of the in-service strategies employed schools as tne 
unit of assignment to treatment condition. Studies of Cross-Age Tutoring 
and School Store used random assignment of individual students to 
condition. Studies of Drug Edjcation involved random assignment of classrooms 
to condi tion . 

Students in the control and experimental groups received a pretest and 
posttest that assessed the following: 

# The impact of each strategy on affective variables that were 
hypothesized to be causally related to substance abuse (see 
Table 2): 

# The impact of the strategy upon specific measures of drug-use-- 
attitudes toward use, intentions to use, lifetime use, and 
current use (see Table 2). 
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Follow-up testing was performed at the end of a second year in some of 
the studies. These follow-up assessments helped co determine whether initial 
effects of the strategy were maintained, or whether there were effects that 
became evident only after some time had passed. 

Outccmes were assessed with the measures listed in Table 3. In Table 3, 
the designation "T" refers to information furnished by teachers, "S" to 
students, and "A" to archival records. Information obtained only in the 
elementary studies is designated "E" and information obtained only in the 
junior high studies is designated "J." All of the measures except one (drug 
knowledge) demonstrated adequate internal consistency reliability. 

The data for each study were subjected to multiple analyses In several 
elementary school studies and in the drug education studies, student data were 
first aggregated into 'zlasses and the class-level data were analyzed or a 
hierarchical analysis was conducted. In the other studies, student-level 
data were analyzed. Separate analyses were conducted for each sex and grade 
level. The treatment groups were compared at pretest by analyses of variance 
to examine potential biases due to initial nonequaval ence and to attrition. 
The groups were contrasted at posttest by analyses of covariance that con- 
trolled for some pretest differences. Additional analyses examined the 
effects of differential exposure to the intervention utilizing multiple 
regression or analysis of variance. 
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Table 2 

Variables Measured . in Studies 
of Individual Prevention Strategies 



STRATEGIES 



OUTCOME VARIABLES 
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Classroom School Environment 



Teacher Attitudes/Satisfaction 
Faculty Cohesiveness 
Affective Teaching Climate 
Attitudes Toward School 



X 
X 
X 
X 



X 
X 
X 
X 



X 
X 
X 
X 



X 
X 
X 
X 



Personal Satisfaction 



Academic Self-Esteem 
Social Self-Esteem 
Attitudes Toward Peers 
Locus of Control 
Academic Achievement 
Attendance 
Behavior Problems 



X 
X 
X 
X 
X 
X 
X 



X 
X 
X 
X 
X 
X 
X 



X 
X 

X 
X 
X 
X 



X 
X 
X 
X 
X 
X 
X 



X 
X 

X 
X 
X 
X 



X 
X 

X 
X 
X 
X 



Perceived Norms/Social Support 

Perceived Peer Attitudes Tov/arH School x x x x 

Perceived Peer Attitudes Tov/ard Drugs x x x x 

Perceived Prevalence of Drug Use x x x x 



A 

X 



X 
X 
X 



X 
X 



Drug Attitudes 

Acceptance of Licit and/or Illicit Use 
Perceived Utility of Drug Use 
Knowledge Regarding Drugs 



X 
X 



X 
X 



X 
X 
X 



X 
X 



X 
X 



X 
X 



X 
X 
X 



Intentions Regarding Lrug Use 



Behavior Regarding Drug Use 



ERIC 



14. 



Table 3: Outcome 

Variable 

Teacher Attitudes 
Teacher Satisfaction 
Faculty Cohesiveness 

Affective Teaching Climate 

Attitudes Toward School 
Academic Self-Esteem 

Social Self-Esteem 
Attitudes Toward Peers 

Locus of Control 
Academic Achievement 

Attendance 
Behavi or Problems 



Variables and Measures 
Measure 

• Project-developed scales measuring role 
importance and role effectiveness (T) 

• Adapted from Purdue Teacher Morale 
Inventory (T) 

• Adapted from Teacher Cooperation Scale 
of the Teacher Attitude and Classroom 
Climate Questionnaire and from Intimacy 
Scale of the Organization Climate 
Description Questionnaire (T) 

• Adapted from Interpersonal Relationships 
with Pupils and Authority and Control 
Scales of the School Sentiment Index and 
Teacher Affiliation Scale of the Self 
Observation Scales (S) 

• Adapted from School Affiliation Scale of 
the Self Observation Scales (S) 

• Adapted from Scholastic Scale of the 
Self-Appraisal Inventory and the Self 
Observation Srales (S) 

• Adapted from Social Confidence Scale of 
the Self Observation Scales (S) 

• Adapted from Peer Affiliation Scale of 
the Self Observation Scales (S, E) 

• Adapted from 1+ and I- Scales of the 
Intellectual Achievement Responsibi li^v 
Questionnaire (S) 

• Total Reading and Total Math Scales of 
the Stanford Achievement Test (A,E) 

• Grade Point Average (A,J) 

• School district and school absenteeism 
records (A) 

• Project-developed items measuring 
frequency and seriousness of classroom 
misbehavior (T,E) 

• School Discipline Records (A,J) 
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Table 3 (Cont.) 



Variable 

Perceived Peer Attitudes 
Toward School 

Perceived Peer Attitudes 
Toward Drugs 

Perceived Prevalence of 
Drug Use 



Acceptance of Licit and 
Illicit Drug Use 



Perceived Utility of 
Drug Use 



Knowledge Regarding Drugs 



Intentions Regarding Drug 
Use 

Behavior Regarding Drug Use 



Measure 

Adapted from eight instruments measuring 
attitude^ toward school (S) 

Project-developed scale measuring the 
degree to which students think their peers 
support the use of drugs (S) 

Projec'c-developed scale measuring the 
degree to which students think their peers 
use different drugs (S) 

Project-developed scales assessing 
attitudes toward selected substances (S) 

Adapted from three instruments ineasuring 
attitudes toward licit and illicit drugs 
(S,J) 

Project-developed scales measuring 
perceived benefits and costs of use of 
selected drugs (S) 

Project-developed scale measuring drug 
Knowledge (S,J) 

Project-developed scale measuring 
anticipated use of various drugs (S,J) 

Project-developed scales measuring lifetime 
use and current use of various drugs (S) 



Note: Letters in parentheses refer to information furnished by teachers (T), 
by students (S), or to archival records (A). Information obtained only 
in the elementary studies (E) is further identified, as is that obtained 
only in the junior high studies (J). 
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RESULTS 

In-Service Strategies 

Between 53% (Jigsaw) and 93^. (Magic Circle) of eligible teachers voluntarily 
enrolled in the in-service training course. Only a few of those enrol le J failed 
to complete the training, and the participating teachers rated all of tne in- 
service courses highly with respect to organization, usefulness, enjoyableness , 
and interest. They rated the trainers as knowledgeable, personable, and effec- 
tive. They consistently attended the training sessions and they participated 
enthusiastical ly. 

Regarding classroom teachers' implementation, the teac!. -s' self-reports 
indicated tnat the skills were proving useful in their classrooms, and that 
they believed they were using the skills proficiently.- With Magic Circle, 
classroom observations shewed that most teachers were able to conduct Circle 
sessions adequately. Teachers' logs indicated that on the average, students 
participated in one Circle per week. Uith ECM, researchers' observations 
failed to shew classroom use of many skills. With Jigsaw, implementation 
averaged two hours per week according to teachers' logs, but observations 
showed that only a third of the teachers applied this strategy without modifi- 
cations that substantially reduced or eliminated peer teaching and inter- 
dependence among students. 

Outcome evaluation results showed that none of the strategies had con- 
sistent effects on teachers' satisfaction with teaching, faculty cohesi veness , 

^An exception was the problem-solving skills taught in the first version 
of the ECM strategies. The teachers found these too complicated and time 
consuming to master or use, and they were replaced with discipline skills m 
the revised versions of the strategies. 
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or teachers' attitude-- toward educational objectives associated with each 
strategy. Nor did the studies shew any of the strategies to have a pattern 
of significant effects on student outcomes. Mone had consistent effects on 
the mediating or the drug-specific variables. Furthermore, no patterns of 
positive effects were revealed by natural variation analyses which examined 
outccxnes for students exposed to relatively high quality and/or quantity of 
implementation as compared with other students. 

Alternatives Strategies 

Process evaluation results shaved that students favorably rated the Cross- 
Age Tutoring and School Store courses as compared with other elective courses. 
Students completing the courses reported that they had learned a lot, that they 
enjoyed the practical experience, and that they believed their efforts had been 
helpful to others. However, tutors tended to dislike their weekly class meetings, 
describing them as irrelevant, repetitive, and boring. Nearly one-fourth of 
the tutors dropped out of the course during the semester. Students in School 
Store reported liking both the daily class sessions and their work in the store. 

All completed the course. In the subsequent Cohort II study the Cross-Aae 
Tutoring course was revised, and both courses v/ere rated highly by students. 

Neither course showed a pattern of significant effects on student outcomes. 
In particular, there was little evidence for enhancement of students' attitudes 
toward themselves or school, as originally anticipated. Follow-up testing 
after one year also failed to show any pattern of effects. 

Druj Education Strategy 

The process evaluation of the initial Drug Education course revealed that 
students and classroom teachers found the course too technical and fast-paced. 
Cour:e ratings from students were mediocre in terms of usefulness, clarity, 
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interest, and enjcyableness. The course was revised for the subsequent studies 
and was well received by classroom .teachers , highly rated by students, and 
apoeared to be more successful at involving students in discussions and exper- 
iential activities. 

The three evaluations of the Drug Education course shcfwed it to have no 
pattern of effects on seventh grade boys, but provided some evidence of positive 
short-term effects on girls, as shown in Table 4. The first two studies showed 
short-term effects on girls' drug knowledge, and their perceptions of peer 
attitudes toward, or use of drugs. Short-term effects on girls' involvement 
in alcohol and marijuana use were also shown in the first study, and on cigarette 
involvement in the second study. However, most of these effects had dissipated 
by the follow-up. Furtherniore, the effects did not replicate in the third 
study, which shewed no effects. Taken together, these results suggest that 
the effects of the courses on girls are at best inconsistent and short-lived. 

Cohort Studies 

Experimental students in the Cohort I study received two years of Magic 
Circle follov/ed by one year of Jigsaw. These students were in the fifth grade 
at the end of the third yea*-. Third-year results showed no difference between 
experimental boys and boys in the control group. Several negative effects 
were found for girls but these probably were due to extraneous factors such 
as initial student differences. 

Cohort II experimental students received ECM-Junior High and Drug Educa- 
tion as seventh graders, and ECM-Junior High as eighth graders. One-third also 
received Cross-Age Tutoring or School Store as eighth graders. Second-year 
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results shewed that as compared to controls, experirnental stiidents had 
greater drug knowledge. Experimental girls were less involved in cigarette 
use and perceived that fewer of their peers were using drugs. Further 
analyses indicated that the effects wersi attributable to the Drug Eiucation 
course (see Table 4 for first-year effects of this course upon Cohort II 
experimental girls). 
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